1
$\begingroup$

I am using Catboost and one thing I notice in the guide is that it says to not preprocess to one-hot encoding.

My data has a single target per row however the feature can have both thousands of values and multiple values associated with the same row. I am struggling how to best present this data to Catboost. Any given row could have zero, one, or twenty of these values associated with it.

My first thought was to use something like 20 'holder' columns and put the feature values into those to associate with the row, however there is no particular order to the feature values; the same value would tend to 'jump' between columns.

My second thought is one-hot, a column per possible feature value. This will create thousands of new features that will each be active only a small percentage of the dataset. I feel like this is the wrong approach esp. since catboost explicitly says to not do this.

My third thought was to duplicate my data such that this feature has a single value per row and duplicate the target sample for each active feature value for that target step. So if I have 10 associated values, I will create 10 rows with the same target and a different value for the feature in each row.

My main confusion is how to handle the 1:many target:feature-value relationship. I have read about 'feature extraction' but not sure if that works in my case. Any given value of the feature is only active a small percentage of the time. Should I just ignore these feature values, despite the fact I know they should have an impact on the target?

There is likely a grouping of values that will cause the same general effect on the target variable, however which values group together to cause this I do not know ahead of time.

Thoughts?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

however there is no particular order to the feature values

That's fine. Just arbitrarily impose one. Lexically ordering them would probably be simplest. Or pre-process your training data to assign rank order labels, with 1 being the most commonly occuring, and always present them to catboost ordered by their rank labels.

likely a grouping of values will cause the same general effect on the target variable, however which values group together to cause this I do not know ahead of time.

Not a problem. Just train an XGBoost model on the side, and examine the resulting Random Forest. The first few decision points in successful trees will reveal the structure. Use that to derive synthetic features which you feed to catboost during training.

Improve this answer
$\endgroup$
7
  • $\begingroup$ "arbitrarily impose one. Lexically ordering" - so in my example, I have say 20 placeholder columns for potential feature values. If record r1 has values "AAA" and "BBB" and "CCC", we would allocate those across placeholder columns 1-3. If r2 has only the latter two values, "BBB" shifts from columns 2 to column 1, as does "CCC". Is this ok? I thought this would confuse the model because the values are 'hopping' volumes between rows. $\endgroup$ Commented May 8, 2024 at 18:09
  • $\begingroup$ The details matter. Show us how you're training and evaluating the catboost model. $\endgroup$ Commented May 8, 2024 at 22:37
  • $\begingroup$ I'll do my best although the problem is that I don't know how to train the model on 1:many feature values per row. I'll try to write some sample code but let me give a mental version of the problem. I want to measure the speed of a school bus on its route. The route speed that day depends on what students get on the bus. Some students take longer to pick up than others, some take about the same. All we know about the students is the roster of names, not their locations, and we have the time the bus took. There could be 1 to 20 kids on the bus any given day. $\endgroup$ Commented May 9, 2024 at 2:29
  • $\begingroup$ What I'm hearing is that "unordered set of student IDs" is almost adequate for the modeling task, but less than ideal. There is no substitute for Subject Matter Expertise. Recommend you spend a day riding along in the bus, or at least discuss it over coffee with a bus driver. We want raw data collection, or post-processing, to identify there were $M$ clumps of $N_{m}$ students at each loading event. And then a model could predict latency for each such event. Sum the event predictions to obtain a bus route total. Knowing day-of-week for various after school events could predict riders. $\endgroup$ Commented May 9, 2024 at 3:07
  • $\begingroup$ That sums it up well. The real problem is actually slightly different as there is no bus driver. :) Each clump of students is likely to correspond to a certain route time. We do not know the individual stats of stops on the route, only the total time for that day's trip. What do you suggest to identify relevant groupings? I know there are techniques like k-means clustering but I am not familiar with them. If I identify the clumps, how do I encode that for catboost, a column per clump? The number of M Clumps is > N students due to various combinations. Thank you for your help! $\endgroup$ Commented May 9, 2024 at 12:55

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.