Add dataloader lib - #5
Merged
Merged
Conversation
Closed
Closed
jonnypress
reviewed
Jul 30, 2025
|
|
||
|
|
||
| This package is used for automated customisable dataloading and database creation and is a generalisation of http://code.kx.com/wiki/Cookbook/LoadingFromLargeFiles. | ||
| Load all delimeted files in a directory into memory in configurable chunk sizes then output the resulting tables to disk in kdb+ partiioned format. |
Contributor
There was a problem hiding this comment.
partiioned - typo
I think this is underselling it a bit. It will not load it all into memory and then to disk. It will load the data chunk by chunk, so the aim is to mininimise memory usage. The memory usage for this should be related to the maximum of
- the space required to load into memory and save one chunk of data
- the memory required to sort the resultant table
So therefore we should be able to load large volumes of on-disk data using a relatively small memory footprint.
A couple of examples would be good:
- loading very large files, but in small chunks
- loading data across partitions from a number of small files e.g. if we had a month worth of AAPL data in one file, and a month of MSFT data in one file, I believe the way this is structured it would handle it relatively efficiently.
Contributor
There was a problem hiding this comment.
Added documentation highlighting the advantages of chunking
To account for the new import method, have removed the top level (.loader) namespacing. Also eliminated the second level .util namespacing as seemed superfluous. Unclear around how setting globals within a namespace from within a function will be impacted by new import changes so have changed how globals are set within the init function.
Account for removing of namespacing in q script; may need to change this again once mechanism for package importing becomes clearer
File which creates private namespace of functionality and exposes public interface
| flip loadparams[`headers]!(loadparams[`types];loadparams[`separator])0:rawdata] | ||
| / loads data in from delimited file, applies processing function, enumerates and writes to db | ||
| / NOTE: it is not trivial to check user has inputted headers correctly, assume they have | ||
| data:$[(`$"," vs rawdata 0)~loadparams`headers; / check if first row matches headers provided |
Member
There was a problem hiding this comment.
as per https://github.com/DataIntellectTech/kdbx-packages/blob/main/style.md please place comments on preceding line
Member
There was a problem hiding this comment.
unresolving this comment as comments are still in-line
jonathonmcmurray
approved these changes
Oct 29, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Same concept as TorQ's/Kx dataloader script.
Tested for kdb+ 4.1 on Linux
Fixes #38