-1

I have a list of lists from a Pandas Data frame that contains nan values. I would like to remove all the lists that contain any nan values. Then plot the remaining values.

Example

test = [[1,2],[1,nan],[3,4]]

End Results

test = [[1,2],[3,4]]

If possible I would also like to make a graph from the end results using any plotting method. Any help you can provide on the manner would be greatly appreciated.

I apologize for the confusion that this question has caused, the list of lists was taken from a pandas data frame hence why nan is present in the example. I would like to keep the data frame as is and would like to work with the list of lists

Panic_Picnic
  • 27
  • 5

2 Answers2

0

Removing values from list

Using filter

nan isnt proper Python syntax and is only used in the Pandas library. So I am going to replace this value with None.

we can use a filter function to remove any values within the list that contains a nan value.

test = list(filter(lambda x: None not in x, test))

Using list comprehension

Edit: Should be x instead of test. Thanks @fountainhead

test = [x for x in test if None not in test]

test = [x for x in test if None not in x]

Both will return a list with values not containing None in them

Plotting

We can use a library called matplotlib to plot a graph. For example, to create a scatter plot:

import matplotlib.pyplot as plt
plt.scatter(x, y)

where plt.scatter takes in x values and y values for row and columns respectively. The full code is as follows:

import matplotlib.pyplot as plt

test = [[1,2],[1,None],[3,4]]
test = list(filter(lambda x: None not in x, test))

x = [x for x,y in test]
y = [y for x,y in test]

# Assume all first elements are x values second elements are y values
plt.scatter(x, y)
plt.show()

The example images is as follows

enter image description here

Anthony Inthavong
  • 344
  • 1
  • 8
  • I don't think `test = [x for x in test if None not in test]` will work, since nan is different from `None`. – fountainhead Nov 06 '20 at 18:36
  • If the value is nan, then we can check if the np.nan from numpy is in the list instead – Anthony Inthavong Nov 06 '20 at 18:38
  • -- It still won't work. There seems to be a typo error in your list comprehension. The expression `[x for x in test if None not in test]` should be corrected to `[x for x in test if None not in x]` – fountainhead Nov 07 '20 at 00:37
  • @fountainhead yes! Thanks for pointing that out, will fix – Anthony Inthavong Nov 07 '20 at 01:02
  • If `np.nan` is present, your `filter()` solution and list-comprehension solutions will both work. If, however, `np.float64("NaN")` or `np.float32("NaN")` or `np.float("NaN")` is present, both your solutions will fail. For detailed reasons, refer [here](https://stackoverflow.com/questions/9904699/checking-for-nan-presence-in-a-container). To make your `filter()` solution work for these cases, the correct lambda should be `lambda x: not any(math.isnan(y) for y in x)`. To make your list comprehension work for these cases, refer my answer to OP. – fountainhead Nov 07 '20 at 05:47
  • In the lambda that I suggested above, you could also use `np.isnan()` in the place of `math.isnan()` – fountainhead Nov 07 '20 at 06:01
0

You could re-construct your list using list comprehension, without the sub-lists having nans:

import math

test = [sub_list for sub_list in test if not any(math.isnan(x) for x in sub_list)]

If you want to delete from the test list 'in-place', you could do this:

import math

for (which, sub_list) in enumerate(test[::-1]):
    if any(math.isnan(x) for x in sub_list):
        del test[len(test)-1-which]
fountainhead
  • 3,584
  • 1
  • 8
  • 17