1
$\begingroup$

I'm working on my first NN following a tensorflow tut and trying to use my own data. After about 80 attempts of formatting my data and trying to load it into a dataset to train I'm throwing the towel.

Here is how my data currently looks

syslog_data = [ [302014,0,0,63878,30,3,1], [302014,0,0,3891,0,0,0], [302014,0,0,15928,0,0,2], [305013,5,0,123,99999,0,3], [302014,0,0,5185,0,0,0], [305013,5,0,123,99999,0,3], [302014,0,0,56085,0,0,0], [110002,4,2,50074,99999,0,4],

In this the last item in each list is the label. If you can tell me if I need to reformat my data and how or just how to get it loaded into a dataset properly.

Thanks for any help or advice you can give

Here is the full code:

import tensorflow as tf import numpy as np from tensorflow.keras import layers from . import syslog print(tf.VERSION) print(tf.keras.__version__) model = tf.keras.Sequential() # Adds a densely-connected layer with 64 units to the model: model.add(layers.Dense(64, activation='relu')) # Add another: model.add(layers.Dense(64, activation='relu')) # Add a softmax layer with 10 output units: model.add(layers.Dense(10, activation='softmax')) model.compile(optimizer=tf.train.AdamOptimizer(0.001), loss='categorical_crossentropy', metrics=['accuracy']) dataset = tf.data.dataset.from_tensor_slices(syslog) model.fit(dataset, epochs=10, steps_per_epoch=30)
Improve this question
$\endgroup$
3
  • $\begingroup$ WElcome to Data Science SE! Which tutorial did you follow? What error are you actually getting? Have you read the Keras documentation? Or the relevant Tensorflow docs for from_tensor_slices? $\endgroup$ Commented Apr 25, 2019 at 19:15
  • $\begingroup$ Ive followed about 15 :/ but this one is the most relevant tensorflow.org/guide/keras I have received a number of errors from different attempts. the most recent is this - got shape [8972], but wanted [8972, 1]. from this code dataset = tf.data.Dataset.from_tensor_slices((data, labels)). Im pretty lost on what my training data should look like and how i should import it. $\endgroup$ Commented Apr 25, 2019 at 19:23
  • $\begingroup$ I can reformat as needed, I just dont know what to do $\endgroup$ Commented Apr 25, 2019 at 19:24

2 Answers 2

Reset to default
2
$\begingroup$

There are a couple of problems and things you might want to add to your existing script.

Below I separate your example data into two NumPy arrays:

  • input values x
  • labels y

It is also important to make sure they are of type float32, because Tensorflow will complain if you pass it integers (as they otherwise would be interpreted).

The following works for me, the model trains to completion:

import numpy as np import tensorflow as tf from tensorflow.keras import layers syslog_data = [ [302014, 0, 0, 63878, 30, 3, 1], [302014, 0, 0, 3891, 0, 0, 0], [302014, 0, 0, 15928, 0, 0, 2], [305013, 5, 0, 123, 99999, 0, 3], [302014, 0, 0, 5185, 0, 0, 0], [305013, 5, 0, 123, 99999, 0, 3], [302014, 0, 0, 56085, 0, 0, 0], [110002, 4, 2, 50074, 99999, 0, 4], ] print(tf.VERSION) print(tf.keras.__version__) x = np.array([arr[:-1] for arr in syslog_data], dtype=np.float32) y = np.array([arr[-1:] for arr in syslog_data], dtype=np.float32) model = tf.keras.Sequential() # Adds a densely-connected layer with 64 units to the model: model.add(layers.Dense(64, activation="relu")) # Add another: model.add(layers.Dense(64, activation="relu")) # Add a softmax layer with 10 output units: model.add(layers.Dense(10, activation="softmax")) model.compile(optimizer=tf.train.AdamOptimizer(0.001), loss="categorical_crossentropy", metrics=["accuracy"]) model.fit(x, y, epochs=10, steps_per_epoch=30)
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I know we just met but I love you $\endgroup$ Commented Apr 25, 2019 at 19:37
1
$\begingroup$ 
import keras import numpy as np full_data = np.array(syslog_data) X = full_data[:,:6] Y = full_data[:,6] # Convert labels to categorical one-hot encoding one_hot_labels = keras.utils.to_categorical(Y, num_classes=10) model.fit(X,Y, epochs=10, steps_per_epoch=30)

Does this work? I think I might be misunderstanding the problem.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I didnt under stand that it needed to be an array, thank you for replying $\endgroup$ Commented Apr 25, 2019 at 19:39

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.