FG Spreadshirt Swag
Page 1 of 5 123 ... Last
  1. #1

    FG Browser with search [CoreRPG, 5e, SWD, other]

    This extension adds a browser-style functionality for the purposes of organizing and searching records and reference library data.

    I made it for my own purposes as I was having a hard time managing all of the campaign records in a long-running game with a lot of recurring NPCs and other campaign records.

    I also wanted a search feature to be able to perform full-text search of all records and library data in a single place, because I would often forget where something was and wanted an easier way to find NPCs based on data in their notes, find story records based on their content, or search the library reference data.

    The example screenshots below are taken from the 5e ruleset just because that is a very popular game system, but the extension should work with any ruleset based on CoreRPG. I myself use it in my own homebrew ruleset.

    Installation

    This is available over on the Forge: FG Browser (with search)

    Or if you prefer, you can manually download the ext file here and install in your Extensions directory.

    Usage

    It can be used in 2 ways, as a campaign record type, or as dynamic browser windows that will not be stored permanently.

    To create a persisted browser type campaign record, it is available under the "Browsers" campaign record type on the sidebar. You can create a blank browser and add records either by drag/dropping them into the tab list, or by using the search page to look up records. Here is an example:



    When using the search page, you can use the filter fields to filter the results by record class, name or the source of the record. You can either left-click on search results to open them in a new tab and immediately switch to the tab, or middle-click to open them in a new tab without switching to it (like most web browsers). Tabs can be closed via the middle mouse button.

    Tabs can also be renamed either by double-clicking the tab, or right-clicking and using the edit option in the radial menu.

    You can also use the browser without linking it to a campaign record. To do so, just use the chat command:

    Code:
    /foogle <search string>
    The optional search string parameter, if provided, will perform a search automatically when opened.

    If you have a dynamic browser window that you want to persist to the campaign data, you can right-click and there is a save option in the radial menu which will copy the dynamic browser to a persisted one. Here is an example of a dynamic browser:




    Update 1.1.0

    Several people reached out to me to ask about improvements to the search functionality (mainly search performance). The initial version of the search was almost an afterthought, so it was pretty basic. The new version adds a search indexer that indexes the campaign and module records on campaign load, so that searches will have better performance. It also implements search scoring to try to sort the results so that the most relevant results are at the top.

    The "Module Indexing" option button can be used to select which modules to index for search, and the "Rebuild Index" button can be used to trigger the indexer to run again if something gets messed up in the search index (or if "Index on-load" is disabled).

    Update 1.2.1

    The search indexing worked ok on very fast computers, but introduced issues in some environments (it was particularly sensitive to read speed on the drive holding the module data). To that end, in 1.2.1 I implemented a background event loop to handle the indexing (since the FG Lua environment doesn't support coroutines).

    In order to work well across the widest possible range of environments, the indexing behavior can be modified to suit your needs:



    To keep things as simple as possible, there are some built-in config profiles accessible via the slash command:

    Code:
    /indexMode <min|low|normal|high|max>
    If you only want to use the browser for tabbing records, and don't care at all about search, /indexMode min will disable indexing on-load entirely, and indexing will only run when you manually click the "Rebuild index" button. So if you want to have the browsers for organizing records, and don't care about search at all, then do that.

    The others just gradually turn up the indexing workload and scheduling priority.

    The main setting that is going to impact the responsiveness of the client UI is the "Event Loop Priority" setting. Lower values of this setting will make the indexing take longer, but will return control to the FG client loop more frequently, and so will make the client feel more responsive while indexing is running. If you notice stuttering or other performance issue with the UI while indexing, you can set this lower. If you want indexing to run faster, and the UI is sufficiently performant for your liking, you can turn it up.

    There is also a blocking setting if you just want indexing to lock up the client and run as fast as it can.

    The default setting for priority is "Automatic" which tries to adjust the scheduling priority dynamically every second to ensure that it is returning control to the FG client main thread at least 30 times / second. Automatic should well in most cases, but you can turn it up/down as desired.

    You can also turn off indexing for non-text fields (number, image, dice, links, etc.) for a slight increase in performance, and you for formattedtext fields for a modest boost. Basic Indexing restricts indexing to simple string fields only, and uses a much simpler lexical tokenizer for a significant boost in indexing performance at the cost of a less thorough search feature.
    Last edited by sirnoobsauce; April 21st, 2023 at 09:10. Reason: Updating for version 1.2.1

  2. #2

  3. #3
    Zacchaeus's Avatar
    Join Date
    Dec 2014
    Location
    Scotland
    Posts
    20,664
    Jolly well done.
    If there is something that you would like to see in Fantasy Grounds that isn't currently part of the software or if there is something you think would improve a ruleset then add your idea here http://fgapp.idea.informer.com/

  4. #4
    Nice idea!

  5. #5
    Quick question. Does this work on the Player side as well?

    I am thinking of notes they make on character sheets, or other extensions that allow the players to to create separate notes (from Mad Nomad for example).
    We do not stop playing because we grow old.
    We grow old because we stop playing.

    www.islandkingdoms.org

  6. #6
    Quote Originally Posted by anstett View Post
    Quick question. Does this work on the Player side as well?

    I am thinking of notes they make on character sheets, or other extensions that allow the players to to create separate notes (from Mad Nomad for example).
    That's a good idea. I went ahead and implemented that. The build is still queued for processing on the Forge, but you can manually get the ext here if you want it now (and should be on the forge within a day, whenever it lets me set it as the live version).

  7. #7
    Absolutely fantastic and essential sirnoobsauce - sent you a msg about your indexing implementation - if you need me to test another alpha build, shoot me a message.

  8. #8
    With today's update, FGU is taking upwards of 8 minutes to load a campaign. It is usually less than 2 minutes.

    Autosave and using "/save" are taking 55-75 seconds instead of the usual 1-2.

    The problem is the same on (relatively) high-end desktop and laptop.

    Return to launcher is taking multiple minutes instead of seconds.

    I waited 10 minutes after finally getting a player connected and was still unable to see any characters to select. Usually they pop up after 30 seconds or so.

    Disabling this extension resolves the issue.

  9. #9
    I can confirm the above issue - we had to turn it off in our game last night. Something else strange I noticed was random freezes, even when no records had changed and all indexing is set to off. It was acting like it was spiked on memory.

    Core rpg - 6 modules loaded - not too much.

    Later i’ll start removing one extension at a time and see if that makes a difference

  10. #10
    I pushed a version to the forge with a quick workaround for this while I look into options to make a more robust solution. I will explain more fully what is going on below, but the short version if you just want it to work and are having performance issues like those described above, go into the options and scroll down to the "Search Options" settings, and I would recommend (for now) setting "Persist Campaign Index" to "Off" and then hit the "Rebuilt Search Index" button. That will take a couple of seconds, but that delay when you rebuild the index is how much time the indexing should add when loading or saving the campaign.

    I made "Off" the default this setting in the build, so it may automatically pick that up and you might not need to change it. The other options can be used to speed up the indexing if you still feel it is too slow. For people who only want to use the extension for the tabbed record containers and don't care about search, then I would recommend setting Basic Indexing to On, and turning the other four settings off. Then rebuild the index. With that done, the index building should be very fast and the impact on load/save should be minimal.

    If you're willing, I would actually appreciate some feedback on how much time they add to building the index, and their impact on how useful the search results are. Because I want to tweak the default settings to give the best experience to people in other use cases. To explain the options:

    Basic Indexing - This has 2 effects. First, it simplifies the lexical tokenizer to only index on plain words. If this is off, the tokenizer will perform basic lexing functions like indexing a record for "attack" and "attacking" if it sees "attacking" in a record. Turning this off speeds up indexing by a fair bit. The second effect is to ignore all non-string fields in records. So formattedtext, number, die, token, etc. fields will all be skipped in indexing.
    Index Formatted Text - This will just skip formattedtext fields. If basic indexing is on, this doesn't really do anything since formattedtext would be skipped anyways, but I found that lexing the formattedtext fields was the heaviest part of indexing by a fair bit, so this will improve performance by a bit, without being quite as extensive as the basic indexing.
    Index Modules - Toggles whether or not to include records from loaded modules
    Index Reference Material - Toggles whether or not to include the reference pages from loaded modules

    Anytime you change a setting, you can use the "Rebuild Search Index" button on the bottom pane of the options window to re-index and see how long it takes. The indexer will also log in the console.log how long the indexing took (in seconds) in a print like this:

    Code:
    [4/7/2023 6:45:31 AM] s'SearchIndexer.buildIndex: ' | #0.78899999999976
    Now as to what was causing the huge performance hits (and the intermittent hangups):

    I wanted to avoid having to build the entire index at every startup, so I decided to cache it in the CampaignRegistry, which will accept lua tables. But the index is very large, 30-100MB in my tests, depending on how many records and how large they are. But in most normal cases, the CampainRegistry is on the order of a few KB, it is just a pickled lua table of key/value pairs. So if I store the search index in my test campaign in the CampaignRegistry, the file (CampaignRegistry.lua in the campaign directory) goes from ~2.3KB to 86MB.

    This wasn't causing an issue on my normal PC, but for testing I tried to run the extension on a slightly older PC and it became totally unusable, as mattvictim and spencerg described. I am suspecting that either the NVMe m.2 drive or the more-modern CPU in my main PC is just fast enough that it covers up the issues of handling the pickling/unpickling of such a large CampaignRegistry file. But regardless, I found significant performance improvements by just turning off the index caching altogether and getting the CampaignRegistry.lua back to a more normal size. I decided to err in the direction of making the indexing a bit leaner by default for now. So the default behavior now is to use Basic Indexing with Index Modules on, and everything else off.

    The massive CampaignRegistry is why saving the campaign takes so long, because the in-memory cached index has to pickled and written to the disk. This delay on saving is also why spencerg was experiencing the intermitted hangups in the UI, as FG will periodically save the campaign while it is running. Anytime that happens, the client will hang for however long it takes to save.

    The issues with the client also seem to be caused by the massive CampainRegistry. It looks like the CampaignRegistry is replicated from the host to the client when they connect. When I chuck in 100MB of search index in there, it has a big impact on that operation. I wasn't seeing that in my testing here because the laptop I was using as a client is on the same network as the PC, so that transfer is super fast. But I ran a wireshark capture while connecting and I did see way more data go over the wire when connecting with the cached search index vs connecting without it. So that probably caused the client issue

    The reason I looked for a way to cache the index in the first place is due to a bit of a limitation in the scripting system in FG, which is the inability to run code asynchronously. From my testing, no matter what you do, any Lua code execution blocks the main UI thread, and hangs the entire application while it is running. So something that would be better to run as a background task like building a search index has to run synchronously with the UI. That is why I wanted to try to cache the index, but looking into the reports here today make me suspect that is not going to be a good approach.

    For now, I will wait to hear back if disabling the caching and tweaking the indexing options gets you into a good state, and then figure out what defaults I should set based on your feedback. If you want to follow up via a DM to let me know how it goes on this version, I would appreciate the feedback.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
STAR TREK 2d20

Log in

Log in