Earlier today, I thought that you'd want List<? extends AbstractInst<? extends IInstType>> to be the data type of insts. It's certainly a data type that will match the object that you're creating, but I doubt whether it's actually what you want in this case. An explanation is in order.
Suppose you have one class that extends another, like PrintWriter extends Writer. That is, every PrintWriter is also a Writer, but there are extra methods (like println) that you can call with a variable of type PrintWriter, but not with a variable of type Writer. (I prefer real examples to the standard Dog extends Animal).
It's important to understand that ArrayList<PrintWriter> is not a subtype of ArrayList<Writer>, although intuitively it seems like it might be. The reason is this. Suppose I have a variable myList, and I write myList.add(new StringWriter()); What type could myList be? It clearly can't be of type ArrayList<PrintWriter>, because a StringWriter is not a PrintWriter. But it could be of type ArrayList<Writer>, because a StringWriter is definitely a Writer and therefore must be able to be added to an ArrayList<Writer>.
Therefore, any ArrayList<Writer> would work OK in the line
myList.add(new StringWriter());
but any ArrayList<PrintWriter> would not. Therefore, an ArrayList<PrintWriter> can't possibly be a special type of ArrayList<Writer>, in the same sense that a PrintWriter is a special type of Writer.
To put it another way, it should be possible to write this
ArrayList<Writer> myList = new ArrayList<Writer>();
myList.add(new StringWriter());
and yet the compiler should somehow prevent us from writing this
ArrayList<Writer> myList = new ArrayList<PrintWriter>();
myList.add(new StringWriter());
Since the second line is clearly OK, it must be the first line that generates the compile error. Indeed it is - you can't assign an ArrayList<PrintWriter> to a variable of type ArrayList<Writer>, because an ArrayList<PrintWriter> just isn't an ArrayList<Writer>. Or as Jon Skeet expressed it at https://stackoverflow.com/a/2745301/1081110, "Awooga awooga".
For all of that, ArrayList<Writer>, ArrayList<PrintWriter> and ArrayList<StringWriter> all seem a little alike somehow. It seems there ought to be some type of variable that could refer to any one of these three. And indeed there is - it's ArrayList<? extends Writer>. But this is an abstract type. You can't instantiate it. You can't write new ArrayList<? extends Writer>() - ultimately an ArrayList<? extends Writer> has to actually be either an ArrayList<Writer>, an ArrayList<PrintWriter> or an ArrayList<StringWriter> (or possibly an ArrayList of some other type of Writer).
This shouldn't be too disturbing to us. After all, List<Writer> is also an abstract type. You can't write new List<Writer>(), because a List<Writer> actually has to be either an ArrayList<Writer>, or a LinkedList<Writer>, or some other type of list of Writer.
Also, a variable of type ArrayList<? extends Writer> isn't always the most useful variable to have, because you can't use it to add things to the list. If myList is of type ArrayList<? extends Writer>, then you can't write myList.add(myWriter); no matter what the type of myWriter is, because the compiler has no way of checking that myWriter is the right sort of Writer for the list.
What you can do with a variable of type ArrayList<? extends Writer> is to get things out of the list. If myList is of type ArrayList<? extends Writer>, then you can write
Writer myWriter = myList.get(0);
because no matter whether myList refers to an ArrayList<Writer>, an ArrayList<PrintWriter> or an ArrayList<StringWriter>, you know that what's in it is some kind of Writer.
So to return to the actual question, where you have
List<AbstractInst<? extends IInstType>> insts = new ArrayList<MIPSInst>();
which initially seems reasonable, since, as you've explained MIPSInst is indeed a subtype of AbstractInst<? extends IInstType>. Or to put it another way, every MIPSInst is an AbstractInst<? extends IInstType>.
Obviously, you could have written
List<MIPSInst> insts = new ArrayList<MIPSInst>();
and been able to add things to the list, and get things out of the list. But you wanted to use some type expression that indicated that any kind of AbstractInst<? extends IInstType> is OK, and that you're only going to use this variable for stuff that would work on any kind of List of AbstractInst<? extends IInstType> objects.
As I have demonstrated with my StringWriter / PrintWriter example, the type that you were looking for is List<? extends AbstractInst<? extends IInstType>>. That encapsulates the fact that this list can be a List of any type of AbstractInst<? extends IInstType>, including MIPSInst.
However, unless you cast this variable to something else, using this type will restrict you to a read-only view of the list. You can get things from the list, but you can't add anything at all, because the compiler has no way of checking that you're adding the right kind of AbstractInst<? extends IInstType>.
In this particular instance, you're creating an empty list. So it's probably not very useful to have a reference to it that lets you get stuff, but not add stuff. So contrary to my earlier comment, it's probably best if you just declare your variable as a List<MIPSInst>, then add things and get things to your heart's content.