All Collections
FAQ's
What Does the Daily Update Do?
What Does the Daily Update Do?

VividCharts refreshes data daily. How does this work?

Kseniya Verenich avatar
Written by Kseniya Verenich
Updated over a week ago

The refresh occurs when a scheduled job called VividDailyUpdate is triggered at 12:00am every night.

  1. Find what assets are available in the instance

    1. Tables

      1. Find which Tables are available based on the Apps purchased.

      2. Query the Tables(sys_db_object) table for each table included with the purchased apps.

      3. Clear out the VividTables table. This table dictates which tables can be read in VividCharts.

      4. For each table available in the instance included in the purchased apps, create a new record in the VividTables table for accessibility through VividCharts.

    2. DB Views

      1. Find which Tables are available based on the Apps purchased

      2. Query the View Table(sys_db_view_table) table for each DB view included with the purchased apps.

      3. Clear out the VividDBViews table. This table dictates which DB Views can be read in VividCharts.

      4. For each DB view available in the instance included in the purchased apps, create a new record in the VividDBViews table for accessibility through VividCharts.

    3. Report Sources

      1. Find which Tables are available based on the Apps purchased.

      2. Query the Report Sources Table(sys_report_source) table for each Report Source reporting on tables available with the purchased apps.

      3. Clear out the VividReportSources table. This table dictates which Report Sources can be read in VividCharts.

      4. For each Report Source available in the instance included in the purchased apps, create a new record in the VividReportSources table for accessibility through VividCharts.

    4. PA

      1. Find which Tables are available based on the Apps purchased.

      2. Query the Indicators Table(pa_indicators) table for each Automatic Indicator reporting on tables available with the purchased apps and Show in Analytics Hub is marked as true.

      3. Query the Indicators Table(pa_indicators) table for each Automatic Indicator reporting on tables available not included with purchased apps, flag them as inactive.

      4. Query the Indicators Table(pa_indicators) table for each Formula Indicator and check if the formula indicator is reporting off an inactive indicator Show in Analytics Hub is marked as true

      5. Query the Indicators Table(pa_indicators) table for each Manual Indicator reporting on tables available with the purchased apps and Show in Analytics Hub is marked as true.

      6. Clear out the VividPASources Table(x_vivid_vc_core_vividpasource). This table dictates which PA Indicators can be read in VividCharts.

      7. For each PA Source available in the instance included in the purchased apps, create a new record in the VividPASources table for accessibility through VividCharts.

  2. Remove Unnecessary Data

    1. Remove Old Versions

      1. Determine if a View version is a “Saved” or “Published” version. If it is not and it is older than 7 days, remove the version from the x_vivid_vc_core_slide_version table.

    2. Remove Orphaned Slide Decks

      1. Query for all slide decks that are not attached to a view.

      2. Delete those slide decks from the x_vivid_vc_core_slides table.

    3. Remove Orphaned Versions

      1. Delete Slide Version records from the x_vivid_vc_core_slide_version if they are not attached to a Slide Deck.

    4. Remove Orphaned Slides

      1. Delete Slide records that do not have a version attached from the x_vivid_vc_core_slide table.

    5. Remove Orphaned Charts

      1. Delete Chart records that do not have a slide attached from the x_vivid_vc_core_chart table.

    6. Remove Orphaned Shapes

      1. Delete Shape records that do not have a slide attached from the x_vivid_vc_core_shape table.

    7. Remove Orphaned Text

      1. Delete Text records that do not have a slide attached from the x_vivid_vc_core_text table.

    8. Remove Orphaned Images

      1. Delete Image records that do not have a slide attached from the x_vivid_vc_core_image table.

    9. Remove Orphaned Static Data

      1. Delete Static Data records that do not have a Chart attached from the x_vivid_vc_core_static_data table.

    10. Remove Ghost Collections

      1. Find all collections on the x_vivid_vc_core_collections table that do not have a creator. This can occur when users who have created a collection are removed from the system.

      2. Check if the collection has been shared with any groups, users, or roles. If not, delete the collection.

    11. Remove Orphaned Permissions

      1. Query the x_vivid_vc_core_view_groups table for any view permissions which have an empty group. If so, remove them.

      2. Query the x_vivid_vc_core_view_roles table for any view permissions which have an empty role. If so, remove them.

      3. Query the x_vivid_vc_core_view_users table for any view permissions which have an empty user. If so, remove them.

    12. Remove Orphaned Collection Connections

      1. Find all View Collection Connections on the x_vivid_vc_core_view_collection_connection table where the view OR the collection is empty and remove those records.

    13. Remove Orphaned Portal Connections

      1. Find all View Portal Connections on the x_vivid_vc_core_view_portal_connection table where the view OR the portal is empty and remove those records.

    14. Remove Old Error Messages

      1. Find all Error Messages on the _vivid_vc_core_error_messages table that are older than 7 days and remove those records.

    15. Remove Old Info Messages

      1. Find all Info Messages on the x_vivid_vc_core_info_messages table that are older than 7 days and remove those records.

    16. Remove Old Queue Results

      1. Find all Queue Results on the x_vivid_vc_core_queue_results table that are older than 7 days and remove those records.

    17. Remove Old Usage Data

      1. Find all Usage Data on the x_vivid_vc_core_data_export table that is older than 3 months and remove those records.

    18. Remove Unneeded Collection History

      1. Remove Failed Collections Older than 30 Days - Find all Collection Histories on the x_vivid_vc_core_collection_history table for records that are older than 30 days and have a status of failed. Remove those records.

      2. Remove Collection Histories stuck in a refreshing status - Find all Collection Histories on the x_vivid_vc_core_collection_history table for records that were started before yesterday and have a status of refreshing. Remove those records.

      3. Remove Collection Histories not attached to a view - Find all Collection Histories on the x_vivid_vc_core_collection_history table for records with nothing in the view field. Remove those records.

      4. Remove Unneeded Collection Histories - Query the x_vivid_vc_core_collection_history table for Collection Histories and group the results by the ‘view’. Delete all records except the newest Collection History for each view.

    19. Remove Garbage Views

      1. Remove Slide Views with a Slide Deck Connection that have no Slide Decks - Query the x_vivid_vc_core_views table using a related list query on the x_vivid_vc_core_view_slides_connection table. If the view is not connected to a slide deck, remove it.

      2. Remove Slide Views with no Slide Deck Connection - Query the x_vivid_vc_core_views table using a related list query on the x_vivid_vc_core_view_slides_connection table. If the view is not connected to a slide deck connection, remove it.

      3. Remove Portal Views with no Portal View Connections not connected to any Portals - Query the x_vivid_vc_core_views table using a related list query on the x_vivid_vc_core_view_portal_connection table. If the view is not connected to a portal, remove it.

      4. Remove Portal Views with no Portal View Connections - Query the x_vivid_vc_core_views table using a related list query on the x_vivid_vc_core_view_portal_connection table. If the view is not connected to a slide deck connection, remove it.

    20. Remove Garbage Summary Template Features

      1. Remove all duplicate Template Feature Groups - Query the x_vivid_vc_core_summary_template_features table. For each summary check for duplicate feature groups. If a duplicate is found, delete it.

  3. Garbage Collection

    1. Delete Views marked for deletion

      1. Query the x_vivid_vc_core_views table for records where delete_ready is true. Remove those records.

    2. Delete Version Templates marked for deletion

      1. Query the x_vivid_vc_core_version_template table for records where delete_ready is true. Remove those records.

    3. Delete Summary Templates marked for deletion

      1. Query the x_vivid_vc_core_summary_template table for records where delete_ready is true. Remove those records.

  4. Refresh Daily Views

    1. Generate a Job ID

    2. Query for records on the x_vivid_vc_core_views table in which the type is not summary or portal, type IS slide or report and there is no summary id. The collection cadence must be set to daily or be empty.

    3. For each view found, log the collection job to the x_vivid_vc_core_collection_history table.

    4. Find the published version of the view, if it exists begin the data refresh.

    5. Query the x_vivid_vc_core_filter_group table for the filter group on the view.

    6. Query the x_vivid_vc_core_chart table for any charts on the published version of the view.

    7. Find the filters for the view by querying the x_vivid_vc_core_filter table for the filter group on the view.

    8. For each chart loop over the available filters to refresh the data options and build any filter override values on the charts.

    9. Delete the existing static data for each chart by removing any records from the x_vivid_vc_core_static_data table in which has a matching chart id.

    10. Call the VividCharts DataEngine to regenerate the data for the chart.

    11. Push the updated static data to the x_vivid_vc_core_static_data table.

    12. Update the x_vivid_vc_core_collection_history table records with our Job ID to a completed status.

  5. Refresh Daily Portals

    1. Generate a Job ID.

    2. Query for records on the x_vivid_vc_core_portal_card table whose card deck’s portal has a collection cadence of daily or null.

    3. For each portal card found, log the collection job to the x_vivid_vc_core_collection_history table.

    4. Query the x_vivid_vc_core_portal table for any portals whose collection cadence is daily or null.

    5. For each portal record returned, regenerate the summary data by the PortalFactory script include.

    6. Update the x_vivid_vc_core_collection_history table records with our Job ID to a completed status.

Did this answer your question?