From 2bad757d67c53a1ebc8e38f4923bb05b1656f906 Mon Sep 17 00:00:00 2001 From: Quinten Steenhuis Date: Sat, 26 Sep 2026 17:36:10 -0400 Subject: [PATCH] Document durable background assembly completion and regeneration --- docs/authoring/combining_interviews.md | 2 +- .../AssemblyLine/background_assembly.md | 308 ++++++++++++------ 2 files changed, 202 insertions(+), 108 deletions(-) diff --git a/docs/authoring/combining_interviews.md b/docs/authoring/combining_interviews.md index f3835f571..182a54ad8 100644 --- a/docs/authoring/combining_interviews.md +++ b/docs/authoring/combining_interviews.md @@ -327,7 +327,7 @@ Review the combined interview to ensure a smooth user experience. 4. **Smoothing shared routing logic:** Adjust text and logic to gracefully handle the combined context. For example, if a child interview assumes the user is only generating that specific form, you may need to open up the choices so the user can dynamically decide to create the form using the interview, upload an existing one, or wait until later. 5. **Handling large document packets:** Combined interviews often generate many documents. - **Remove previews:** Remove document previews (e.g., `${ al_court_bundle.as_pdf(key='preview') }`) from signature screens. Generating a preview of a massive packet can be very slow or break the layout. Instead, use a simple confirmation message like *"Review your answers before you sign"*. - - **Use background assembly:** If your interview generates more than ~3 documents, you should switch to background processing to prevent the server from timing out. Update your final download screen to use `al_user_bundle.download_list_html(use_previously_cached_files=True, include_full_pdf=True)`. See the [background assembly documentation](../components/AssemblyLine/background_assembly.md) for more details. + - **Use background assembly:** If generating your combined packet is slow enough to risk a timeout, use background processing. After collecting all attachment inputs, reference `al_user_bundle.downloads_with_docx_ready` in your interview order before the download screen. On that screen, use `al_user_bundle.download_list_html(use_previously_cached_files=True, include_full_pdf=True)` to display the saved files. See the [background assembly documentation](../components/AssemblyLine/background_assembly.md) for more details. 6. **Overriding attachment blocks:** Keep in mind that you can also override an attachment block (e.g., an `attachment:` block that generates a PDF) directly in the parent umbrella interview. Because Docassemble uses the last loaded block with a matching variable name, you aren't forced to make the included YAML totally reusable without changes. You can let the child interview define the base attachment block, and simply redefine it in your `main_` YAML if the combined context requires different formatting or logic. 7. **Document integration points:** It is crucial for the author to document the integration points of the child (included) interview. Clearly explain in comments or an external `README` what specific variables need to be set—especially those that might not be standard AssemblyLine variables—for the child interview to work properly when included in a parent interview. 8. **Test:** Run the main interview and verify that the sections flow logically, there are no missing variables, and progress bars or navigation menus behave smoothly. diff --git a/docs/components/AssemblyLine/background_assembly.md b/docs/components/AssemblyLine/background_assembly.md index 484dcfc81..a438a02c8 100644 --- a/docs/components/AssemblyLine/background_assembly.md +++ b/docs/components/AssemblyLine/background_assembly.md @@ -7,165 +7,259 @@ sidebar_label: | slug: background_assembly --- -Assembly Line `3.2.0` and later introduce a background processing option to the [ALDocumentBundle](al_document.md) -class. +Background assembly lets an [ALDocumentBundle](al_document.md) generate files while +the user sees a waiting screen. Use it when document generation or PDF conversion +is slow enough to risk a web request timeout, especially for large packets or +uploaded exhibits. Small, fast interviews can continue to assemble documents in +the foreground. -Background processing: +Collect all answers needed by the attachments before starting background assembly. +A background task cannot ask the user a question. A review screen before assembly +helps users finish their changes before the files are generated. -* Is safer when your user may generate large documents that take more than 60 seconds to assemble or convert to PDF -* Prevents frustrating 504 gateway timeout errors +## Adding background assembly -Negatives include: +Use a saved completion variable in your interview order. AssemblyLine starts the +task, shows the waiting screen, and continues when the task's callback has saved +the files to the interview. -* Requires more care from the developer -* Usually prevents you from using an "edit" button on the final screen -* Means assembling documents will always take at least 10 seconds +| Output | Completion variable | +| --- | --- | +| Final PDFs | `al_user_bundle.downloads_ready` | +| Final PDFs and editable DOCX files, where available | `al_user_bundle.downloads_with_docx_ready` | +| Preview PDF | `al_user_bundle.preview_ready` | -If your users currently have to wait a long time to get their finished documents, or if your users -ever see an error screen because the downloads took too long to make, you should use background processing. +### Interview order and download screen -If you have only a few small documents and there is no user-provided content (like photos or -other uploaded exhibits), you should try without using background processing first. +After your existing question, review, and signature blocks, reference the +completion variable immediately before the download screen: -## Adding background processing to your interview +```yaml +mandatory: True +code: | + # Keep your existing logic to collect all attachment inputs above these lines. + al_user_bundle.downloads_with_docx_ready + download_screen +--- +event: download_screen +question: | + Download your documents +subquestion: | + ${ al_user_bundle.download_list_html(use_previously_cached_files=True, include_full_pdf=True) } +``` -Background processing is a built-in feature of the ALDocumentBundle class. Using it involves: +For PDF-only generation, replace `downloads_with_docx_ready` with `downloads_ready`. +Generating DOCX files does not require you to display them: the download table +shows PDFs by default. Add `format="docx"` to `download_list_html()` to offer the +editable versions where available. -1. Starting the built-in background assembly `task` in your interview order block -1. Setting conditions on when to display a waiting screen -1. Editing the download screen so it makes use of the background assembled documents +Keep `use_previously_cached_files=True` on the download screen. This tells +`download_list_html()` to use the files saved by the background task. Without it, +the download screen can assemble documents again in the foreground. -In most cases, when you use background processing, you should also: +:::warning Define attachment inputs first -1. Disable the preview screen -1. Show a review screen in-line before the download screen. +Every variable needed to generate your documents must be defined before you +reference the completion variable. This includes variables used to decide which +documents are enabled. `skip undefined` can allow an attachment to omit undefined +values, but it does not collect missing answers or satisfy other assembly logic. +::: -### Editing your interview order block +### Optional preview -Find the line in your interview order block where you would like to start the background -assembly. In most cases, this should be just before the download screen. +A preview uses the same pattern. Include it when users need to inspect the actual +document before continuing; omit it when a review of their answers is enough. +Preview generation adds another wait. -Add a line like this: +```yaml +mandatory: True +code: | + # Collect the inputs required for the preview first. + al_user_bundle.preview_ready + preview_screen + # Keep any signature or other final-answer blocks here. + al_user_bundle.downloads_with_docx_ready + download_screen +--- +continue button field: preview_screen +question: | + Preview your documents +subquestion: | + ${ al_user_bundle._preview_file } +``` + +Use this interview order in place of the earlier one, with the same download +screen. For a runnable example with attachments, see +[`test_aldocument_background_assembly.yml`](https://github.com/SuffolkLITLab/docassemble-AssemblyLine/blob/main/docassemble/AssemblyLine/data/questions/test_aldocument_background_assembly.yml). + +## Why the completion variable matters + +Celery keeps task results temporarily, normally for one day. After the result +expires, a completed task's `.ready()` method can return `False` again. An +interview order that repeatedly calls `.ready()` can then send a returning user +to a waiting screen indefinitely, even though their documents were already saved. + +AssemblyLine's completion variables check the files saved to the interview by the +background callback. They save `True` once those files exist and remain usable +after Celery's result expires. If the user leaves while generation is running, +the callback still saves the files. The completion block recognizes them when +the user returns, even if the task result expired before that first return. + +### Updating an older interview + +Replace this condition in your interview order: ```python - if not al_user_bundle.generate_downloads_with_docx_task.ready(): - al_download_waiting_screen +if not al_user_bundle.generate_downloads_with_docx_task.ready(): + al_download_waiting_screen ``` -:::warning Make sure all required variables are defined +with: -Background assembly requires that all of the logic required to assemble your documents -was triggered **before** the background processing starts. You cannot have any -undefined variables that are needed for the document to generate. +```python +al_user_bundle.downloads_with_docx_ready +``` -Note that if you use `skip undefined` this isn't usually a problem. -::: +Use `downloads_ready` for the PDF-only task or `preview_ready` for the preview +task. Keep the download screen's `use_previously_cached_files=True` setting. +Existing sessions with files saved by the standard callback can use the new +completion variables without rerunning the expired task. Updating AssemblyLine +alone does not replace `.ready()` conditions in your interview YAML. -In some cases, you may want to start the background assembly earlier than just before the download screen. -For example, you may want to show the user instructions, or get information for a task -that will not change the documents, such as eFiling. +## Regenerating after edits -### Editing the download screen +After a user changes an answer, the saved files still contain the previous +answers. Complete the edit workflow and collect any newly required attachment +inputs, then explicitly reconsider the corresponding task: -On the download screen, edit the reference to `download_list_html()` to let it use -the background assembled documents. Using cached files is disabled by default. +| Output to regenerate | Task to reconsider | +| --- | --- | +| Final PDFs | `al_user_bundle.generate_downloads_task` | +| Final PDFs and DOCX files | `al_user_bundle.generate_downloads_with_docx_task` | +| Preview PDF | `al_user_bundle.generate_preview_task` | + +For example, an action can regenerate the final files: ```yaml ---- -id: download -event: download_screen -question: | - Download the documents -subquestion: | - Your documents are ready for download. +event: regenerate_downloads +code: | + reconsider("al_user_bundle.generate_downloads_with_docx_task") +``` - If you need to make any changes, click the "Undo" button and click the button to edit - the answer you want to change. +Link to that action from a completed download screen with: - ${ al_user_bundle.download_list_html(use_previously_cached_files=True, include_full_pdf=True) } +```mako +${ action_button_html(url_action('regenerate_downloads'), label='Regenerate documents') } ``` -### And don't forget +Starting the task clears the saved download files and completion variables. The +interview order reaches `downloads_with_docx_ready` again and waits for the new +files. Reconsidering the preview task similarly clears `_preview_file` and +`preview_ready`. If your preview screen uses a continue-button variable such as +`preview_screen`, undefine that variable too when you want to show the screen +again. + +Reconsidering **only** a completion variable rechecks the current saved files; it +does not regenerate them. Do not reconsider the task on every pass through a +mandatory block, because that would start another task on each reload. -* Remove the preview screen. A waiting screen before the preview isn't a good user experience for most interviews. -* Add the review screen to your interview order block. The user will not be able to click the "edit answers" button - on the download screen. +Use one final-download mode per bundle at a time, and let an existing task finish +before starting another for the same bundle. The PDF and DOCX modes share the +saved download files. Starting either mode clears both download completion +variables and the other mode's task handle. -### Customizing the user experience +If your interview also assembled attachments in the foreground before an edit, +invalidate those attachment variables and any document or bundle caches as part +of your edit workflow. Restarting a background task clears its saved output; it +does not clear every foreground attachment cache. An Undo or edit button alone +is not enough to refresh background-generated files. -#### What happens in the background task +## Customizing background assembly -You may want to provide a non-default set of options to the file generation step. +### Generation options -Override `al_user_bundle.generate_downloads_with_docx_task` or define a task with your -own name. The default contents of this task are: +To change the files generated by the standard DOCX task, override the +`create_downloads_with_docx` event. Keep the standard task-start block so that +regeneration still clears its saved output and completion variables. For example, +this bundle-specific override omits the ZIP while preserving the combined PDF: ```yaml ---- -generic object: ALDocumentBundle -event: x.create_downloads_with_docx +event: al_user_bundle.create_downloads_with_docx code: | - download_response = x.get_cacheable_documents(key="final", pdf=True, docx=True, include_full_pdf=True) - background_response_action(x.attr_name('save_downloads'), download_response=download_response) + download_response = al_user_bundle.get_cacheable_documents( + key="final", pdf=True, docx=True, include_zip=False, include_full_pdf=True + ) + background_response_action( + al_user_bundle.attr_name("save_downloads"), + download_response=download_response, + ) ``` -#### The waiting screen +The PDF-only task uses `create_downloads`. Both events send their results to +`save_downloads`, which persists `_downloadable_files` in the interview. +`get_cacheable_documents()` returns the per-document file information together +with the optional ZIP and combined PDF. Match the options on your download +screen to the files you generate; for the example above, add `include_zip=False` +to `download_list_html()`. -Override `al_download_waiting_screen`, or define a screen with your own name. The default contents -of this question block are: +The preview task uses `generate_preview_event` and sends its result to +`save_preview`, which saves `_preview_file`. These saved outputs are what the +completion blocks check, so preserve the relevant callback when customizing the +standard workflow. -``` ---- -id: waiting screen +### Starting work before the waiting screen + +You can reference `al_user_bundle.generate_downloads_with_docx_task` earlier in +your interview order to start generation while the user reads instructions or +completes an unrelated step. All attachment inputs must already be defined and +must stay unchanged while that task runs. Reference +`al_user_bundle.downloads_with_docx_ready` before the download screen to wait if +generation has not finished yet. + +### Waiting screens + +Override `al_download_waiting_screen` to change the final-document waiting screen, +or `al_preview_waiting_screen` for previews. Keep `reload: True` so the interview +periodically checks for the saved files: + +```yaml +event: al_download_waiting_screen question: | Please wait while we make your documents subquestion: | This can take a few minutes. + -event: al_download_waiting_screen reload: True ``` -The default screen uses [standard Bootstrap components](https://getbootstrap.com/docs/5.2/components/spinners/) -to create the spinning circle. You can put anything you like here, although an animation is recommended. - -If you would like the screen to reload more often than every 10 seconds, [you can set it as low as 4 seconds](https://docassemble.org/docs/modifiers.html#reload). - -## Key methods and built-in task names - -`al_download_waiting_screen` is a Assembly Line provided example of a generic waiting screen. - -`generate_downloads_task` is the name of a [background task object](https://docassemble.org/docs/background.html#background_action) defined -in the standard includes as an attribute of each ALDocumentBundle. It triggers a task to produce a full PDF of the bundle using the `final` -key, a full ZIP, and it will not generate DOCX versions of the items, only downloadable (PDF) versions. - -`create_downloads` is an event defined in the Assembly Line standard includes as an attribute for each ALDocumentBundle. -This event is triggered by `generate_downloads_task`. - -`generate_downloads_with_docx_task` is an alternative to `generate_downloads_task` that does produce the DOCX -as well as PDF version of each document. You can safely use this even if you do not always want to display -the DOCX on the download screen. - -In turn, `generate_downloads_with_docx_task` triggers the event `create_downloads_with_docx` - - - -[get_cacheable_documents()](https://github.com/SuffolkLITLab/docassemble-AssemblyLine/blob/76ead20669674d1a458c15af99aeba44899866cd/docassemble/AssemblyLine/al_document.py#L1826) is a new method that generates -all **enabled** documents in the bundle and returns a list, where each item is a tuple of the DOCX and -PDF versions of the document. - -Optionally, it will also produce a .ZIP and a .PDF version of the full bundle. +The default reload interval is ten seconds; this is a polling interval, not a +minimum assembly time. You can [set an explicit reload interval](https://docassemble.org/docs/modifiers.html#reload) +as low as four seconds. -This is called from a YAML block in the Assembly Line standard include files to define the `_downloadable_files` -attribute of a bundle. +### Custom tasks with different results -[download_list_html()](al_document.md#download_list_html) has an optional parameter, -`use_previously_cached_files` which must be set to `True` to make use of the `_downloadable_files` -attribute of the bundle. +For a task that does not use the standard file callbacks, follow the same pattern: +start the task once, save its result with `background_response_action()`, and use +an intermediate completion block that waits until the saved result is defined. +Do not save a one-time `False` value as the completion variable while waiting. +When explicitly restarting the task, clear both the saved result and the +completion variable. See docassemble's +[background action documentation](https://docassemble.org/docs/background.html) +for how to persist results with a response action. ## Troubleshooting -If your background process has an error, the details of the error (typically a missing or undefined variable) -will be found in the [`worker.log` file](https://docassemble.org/docs/admin.html#logs) \ No newline at end of file +If a returning user is stuck on a waiting screen, check for a direct `.ready()` +condition in the interview order and replace it with the appropriate completion +variable. On docassemble versions that expose `celery result retention days`, +increasing that setting only postpones this problem; the interview should rely +on saved results regardless of the retention period. + +If a new task never saves its files, inspect +[`worker.log`](https://docassemble.org/docs/admin.html#logs). Missing attachment +inputs or other generation errors can prevent the callback from running. A +completion variable does not turn a failed task into a successful one. Correct +the underlying error, then explicitly reconsider the task to retry.