Skip to main content
Data Science

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

7
  • $\begingroup$ "arbitrarily impose one. Lexically ordering" - so in my example, I have say 20 placeholder columns for potential feature values. If record r1 has values "AAA" and "BBB" and "CCC", we would allocate those across placeholder columns 1-3. If r2 has only the latter two values, "BBB" shifts from columns 2 to column 1, as does "CCC". Is this ok? I thought this would confuse the model because the values are 'hopping' volumes between rows. $\endgroup$ Commented May 8, 2024 at 18:09
  • $\begingroup$ The details matter. Show us how you're training and evaluating the catboost model. $\endgroup$ Commented May 8, 2024 at 22:37
  • $\begingroup$ I'll do my best although the problem is that I don't know how to train the model on 1:many feature values per row. I'll try to write some sample code but let me give a mental version of the problem. I want to measure the speed of a school bus on its route. The route speed that day depends on what students get on the bus. Some students take longer to pick up than others, some take about the same. All we know about the students is the roster of names, not their locations, and we have the time the bus took. There could be 1 to 20 kids on the bus any given day. $\endgroup$ Commented May 9, 2024 at 2:29
  • $\begingroup$ What I'm hearing is that "unordered set of student IDs" is almost adequate for the modeling task, but less than ideal. There is no substitute for Subject Matter Expertise. Recommend you spend a day riding along in the bus, or at least discuss it over coffee with a bus driver. We want raw data collection, or post-processing, to identify there were $M$ clumps of $N_{m}$ students at each loading event. And then a model could predict latency for each such event. Sum the event predictions to obtain a bus route total. Knowing day-of-week for various after school events could predict riders. $\endgroup$ Commented May 9, 2024 at 3:07
  • $\begingroup$ That sums it up well. The real problem is actually slightly different as there is no bus driver. :) Each clump of students is likely to correspond to a certain route time. We do not know the individual stats of stops on the route, only the total time for that day's trip. What do you suggest to identify relevant groupings? I know there are techniques like k-means clustering but I am not familiar with them. If I identify the clumps, how do I encode that for catboost, a column per clump? The number of M Clumps is > N students due to various combinations. Thank you for your help! $\endgroup$ Commented May 9, 2024 at 12:55