Rendered at 11:27:05 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
gavinray 24 hours ago [-]
If this actually works, this is such a huge boon. Let me share why:
I work on connectors to various datasources, mostly SQL DB's
The most useful tests are the integration tests, sending actual queries to the source
For most DB's we have a giant Docker Compose which runs a local instance
But things like Snowflake, Databricks, Redshift, Athena suck to test because we need an external hosted instance
I would kill to be able to run test versions of all of these locally
skeeter2020 22 hours ago [-]
Why are you writing tests on the actual provider? IME these types of tests are slow, brittle and don't give you much value. Even if you get a true positive, what can you do about it?
gavinray 20 hours ago [-]
Because those tests are the only ones that prove the connector is working as intended
Our connectors receive a generic query IR from a Datafusion-based engine
Each connector is then responsible for faithfully translating the query IR into the semantically equivalent query for the underlying data source
For a SQL DB this would be a SQL query, for MongoDB a JSON predicate query, for csv/json files it would be an in-memory predicate engine, etc
Running the generated query against the target data source and asserting that this doesn't throw an error and that the records returned are both correct and in the right order is the true test
dionian 21 hours ago [-]
I assume integration tests.
viccis 18 hours ago [-]
When it comes to platforms like Databricks, integration tests are honestly the only useful ones. Grug remains vindicated as always.
tomnipotent 20 hours ago [-]
For cloud databases, in your use case do you care that the queries actually execute or only that they parse? I with every cloud service had their own version of LocalStack/Moto, and while not perfect they've really helped to improve test coverage.
gavinray 19 hours ago [-]
Unfortunately they have to actually execute and have identical semantics to the target source
I recall seeing a project based on DuckDB parser and engine which emulated dialects for one of these but because the sortinh and other behavior is different it wasn't usable
efromvt 24 hours ago [-]
[dead]
akbarian 24 hours ago [-]
Getting tired of the standard AI design, all websites look the same now
smashed 23 hours ago [-]
I would say go straight to their GitHub but there again, same thing.
With a list of "Known gaps" and implemented engines listed with status like "Real", and of course the Quick start first, as Claude loves to do.
It's not "wrong", it just shows low effort. But then again it's just something someone needed and shared it for free so let's not complain too much.
All the products and saas launching with obvious AI landing pages, that the authors did not even care to trim down, that's another story.
pjmlp 24 hours ago [-]
Before it was all those common template sites.
sodastar 26 minutes ago [-]
i actually get the same feeling. There are blatant bad AI landing pages with the same card design we've seen so many times already, but its not like we are coming from the great age of creative landing pages by small companies. Especially the wordpress world where every mom&pop shop gets their site hosted is still miles behind the standard claude landing page in terms of design.
ngruhn 1 days ago [-]
You should call it "pond".
lysecret 22 hours ago [-]
Want to hijack this someone made experience recently with https://ducklake.select/ ? We need to setup a warehouse soon and this looks enticing.
switchbak 16 hours ago [-]
What I don't understand is why they didn't build this project on top of DuckLake instead - from my shallow analysis they vibe coded up a fake Databricks in Python.
Not that I'd use it, I want to avoid Databricks - the Duck ecosystem is far more compelling for my modest needs, and handles local develop a lot better (IMO).
denom 20 hours ago [-]
Ducklake works well. So far it is still under active development (post AWS acquisition). There is support for local disk-based and remote S3 based datalakes, so integration tests become really easy. Seems stable and all the bugs I've found have already been patched.
lysecret 17 hours ago [-]
Any risks in terms of using it on gcp/azure?
viccis 18 hours ago [-]
Praying for the poor Ducklake employees at AWS to keep whatever demon works there at bay who insists that every single data product has to be bolted onto AWS Glue (and its various Data Catalog, Data Lake, etc. partners in crime) in the most clumsy way possible. And please keep it away from Athena, possibly the worst major query engine in terms of support for query features and file formats. I don't want to see AWS-era Ducklake become another failure like S3 Table Buckets are (imo at least).
Zizizizz 21 hours ago [-]
I'd personally love to try this in earnest as well
asabla 21 hours ago [-]
Oh wow! This would've been so useful at my previous work place. I'll save this one for later. Looks pretty decent too for homelabs!
I wonder if there is any plans on extending it further? (just a bit)
isoprophlex 1 days ago [-]
Great for CI i guess; or is this meant as a cost savings measure where you can swap out your databricks for self-hosted this?
viccis 18 hours ago [-]
I'm tempted to use this but the rub with Databricks is that there are always a bunch of different restrictions on what features work in what products (for example RT//Lakehouse looks great but runs a more limited regex parser than other warehouses), and so I'm basically forced to rely heavily on integration tests to catch weird behavior. This would need to copy it closely enough to catch those.
The other problem is that Databricks renames everything like every 6 months so keeping up with them is non-trivial. Their DLT product alone has gone on a jazz odyssey of renames.
dominotw 22 hours ago [-]
Wtf is databricks now anyways? Looks like their ceo is hawking cheap chinese models and some silly harness called ominiagent. Anyone use ominiagent?
I thought they were building their own models at somepoint? what happened to that.
Also looks like they did one full round of english alphabet and are on series L second time around. I feel like we might have some a tech theranos on our hands here.
NDizzle 22 hours ago [-]
Databricks is a furnace. It burns money.
viccis 18 hours ago [-]
Managed Spark. That is still what they are, all the other nonsense aside.
Spark is such a headache to run that people would rather pay Databricks to do it for them. They also have some secret sauce like Photon (C++ version of the Spark engine) and the new RT Lakehouse warehouses (10x or so speed up on read-only work flows in my experience) as well as good proprietary optimization features like Liquid Clustering that keep your Parquet files nicely sized and indexed. Their AI features are terrible and I can't tell you how many Zoom calls I've joined to rescue people from wild goose chases their model took them on.
Their main competitor is Snowflake, which imo has nicer UI/UX and snappier SQL execution, but the trade off is that Snowflake has a bit more lock-in, whereas Databricks is easier to kind of just build on top of your existing infrastructure. Snowflake's AI features are also much better in my experience.
>I feel like we might have some a tech theranos on our hands here.
Based on what I've seen some customers paying them, they are absolutely making heaps of money. A "small" SQL warehouse cluster is $8.40 an hour (~$73k a year if you leave it on) and the default size of X-Large is $56 an hour or $490k a year. It goes up to 5X-Large at $729 an hour or $6.3M a year. So they do absolutely let people with horribly optimized data just pay their way out of dealing with the problem, which is something I've seen multiple times.
switchbak 16 hours ago [-]
They were early movers and established a lot of the patterns in the space, but I just don't see them holding on to this lead. Spark/Hadoop is really so damn inefficient that something like DuckDB shows where that overhead is, and it's extreme in many cases.
I work on connectors to various datasources, mostly SQL DB's
The most useful tests are the integration tests, sending actual queries to the source
For most DB's we have a giant Docker Compose which runs a local instance
But things like Snowflake, Databricks, Redshift, Athena suck to test because we need an external hosted instance
I would kill to be able to run test versions of all of these locally
Our connectors receive a generic query IR from a Datafusion-based engine
Each connector is then responsible for faithfully translating the query IR into the semantically equivalent query for the underlying data source
For a SQL DB this would be a SQL query, for MongoDB a JSON predicate query, for csv/json files it would be an in-memory predicate engine, etc
Running the generated query against the target data source and asserting that this doesn't throw an error and that the records returned are both correct and in the right order is the true test
I recall seeing a project based on DuckDB parser and engine which emulated dialects for one of these but because the sortinh and other behavior is different it wasn't usable
With a list of "Known gaps" and implemented engines listed with status like "Real", and of course the Quick start first, as Claude loves to do.
It's not "wrong", it just shows low effort. But then again it's just something someone needed and shared it for free so let's not complain too much.
All the products and saas launching with obvious AI landing pages, that the authors did not even care to trim down, that's another story.
Not that I'd use it, I want to avoid Databricks - the Duck ecosystem is far more compelling for my modest needs, and handles local develop a lot better (IMO).
I wonder if there is any plans on extending it further? (just a bit)
The other problem is that Databricks renames everything like every 6 months so keeping up with them is non-trivial. Their DLT product alone has gone on a jazz odyssey of renames.
Also looks like they did one full round of english alphabet and are on series L second time around. I feel like we might have some a tech theranos on our hands here.
Spark is such a headache to run that people would rather pay Databricks to do it for them. They also have some secret sauce like Photon (C++ version of the Spark engine) and the new RT Lakehouse warehouses (10x or so speed up on read-only work flows in my experience) as well as good proprietary optimization features like Liquid Clustering that keep your Parquet files nicely sized and indexed. Their AI features are terrible and I can't tell you how many Zoom calls I've joined to rescue people from wild goose chases their model took them on.
Their main competitor is Snowflake, which imo has nicer UI/UX and snappier SQL execution, but the trade off is that Snowflake has a bit more lock-in, whereas Databricks is easier to kind of just build on top of your existing infrastructure. Snowflake's AI features are also much better in my experience.
>I feel like we might have some a tech theranos on our hands here.
Based on what I've seen some customers paying them, they are absolutely making heaps of money. A "small" SQL warehouse cluster is $8.40 an hour (~$73k a year if you leave it on) and the default size of X-Large is $56 an hour or $490k a year. It goes up to 5X-Large at $729 an hour or $6.3M a year. So they do absolutely let people with horribly optimized data just pay their way out of dealing with the problem, which is something I've seen multiple times.