Quick way to import csv file to SQLite for CPA analysis

I want to use CPA to classify objects. My data are normalized after CellProfiler analysis so it is not feasible to use the SQLite file generated by CellProfiler. My csv file is 6 GB large, so I can’t use it directly. I am trying to use pandas and chunksize to import csv file to SQLite database, but it is very slow. Any suggestions on how to do it efficiently? Thank you in advance.

My guess is that doing it with pandas is probably the way to go, but you could always try an SQLite interaction tool like DBBrowser (which can import CSVs as tables) to see if that’s any more efficient.

Within python, our lab has also used odo in the past successfully as part of our cytominer-database package, but as a warning it hasn’t had a new release in a while.

Thank you for your reply!
Btw, when I use CPA to open my database and retrieve cell images, it is extremely slow comparing to using the database generated by CP. Do you have a similar experience? Is it possible that it is because I miss something important when generating the database file?

Are you indexing your database before opening in CPA? Particularly for a large database, that could definitely cause this issue.