I'll give you an overview of what tests should look like, but note that it might not be easy to achieve this without having a certain level of experience, and that it may be better to think of this as of something to work towards over time, rather than something to try an achieve immediately.
Think of test cases as testing (or exercising), in an isolated way, a particular scenario that's of interest to you. You'll have multiple such scenarios for the same unit under test (doesn't have to be a method). However, you have to think about these scenarios a bit more abstractly than just all the possible combinations of inputs and outputs, or you'll get bogged down, and your tests will tend to break easily and be a pain to maintain.
The basic idea is, for each scenario, you do "under these conditions, when I do this, this should happen" - the good ol' arrange-act-assert (or given-when-then).
So, what should these scenarios be? The details will depend on the problem, but generally, try to test the rules, rather than specific outputs (again, it might be hard to come up with what these should be). Each such scenario should also be rather specific (within reason, and to the extent your design will allow), so that when a test fails, you know exactly what the problem is (at least in terms of deviation from expected behavior in a particular scenario), as opposed to just the generic "something about the method doesn't work anymore".
For example, you might want to test the rule that says that encrypting a message, and then decrypting the result reproduces the original message. In such a test, you shouldn't really care what the encrypted message is. You don't care if it's all lowercase, uppercase, if it contains numbers or symbols, or whitespace, if it's meaningful or gibberish. You just want your implementation to not throw, and the message to be recoverable.
@Test void testEncryptDecrypt() { String message = "how much wood can a woodchuck chuck until the woodchuck has chucked enough wood"; String result = Vigenere.decryptVariant("key", Vigenere.encryptVariant("key", message) ); assertEquals(message, result); }You can then parameterize this test to use different messages and different keys - it should pass for all of them. In fact, it should even pass for different encryption algorithms (there's a clear example of LSP for you).
Since you're now splitting test scenarios by rules (the abstract "behavior"), you might let this test cover lowercase, mixed-case, uppercase input strings, or you might organize things differently (tests are code, so you can extract a shared method and call it from multiple tests). You're the designer, so really how the tests are organized and how these rules are conceptualized is up to you.
Some of your other tests are good in that they actually test a rule, rather than check for a specific output, like the test below.
@Test void testKeyEncryption_AnyCase() { String message = "A simple variant is to encrypt by using the Vigenère decryption method and to decrypt by using Vigenère encryption."; final String ALL_CAPS_KEY = "CRYPTO"; final String LOWER_CASE_KEY = "crypto"; final String MIXED_CASE_KEY = "Crypto"; assertEquals(Vigenere.encryptVariant(ALL_CAPS_KEY, message), Vigenere.encryptVariant(LOWER_CASE_KEY, message)); assertEquals(Vigenere.encryptVariant(LOWER_CASE_KEY, message), Vigenere.encryptVariant(MIXED_CASE_KEY, message)); }From reading the test, it looks to me that this checks if the key is treated as case-insensitive, by testing if encrypting the message produces the same result every time. That is en example of a rule. I think the name of the test doesn't communicate that intent particularly well though - I'd rename it to something like testEncryptionKeyIsCaseInsensitive. In fact, since test methods aren't called from elsewhere in the codebase and since their sole purpose is to communicate to you what the rule is and what the test is all about, you can apply different naming conventions to them. E.g., you could just make it into a sentence: encryption_key_should_be_case_insensitive. The naming convention police is not coming to get you.
Some behaviors might be of special interest, like how your system under test handles specific edge cases, so I'd probably create those as separate tests. Think of tests as specifying the behavior: what should happen when the input is null? What when it's empty? What when it's all whitespace? Then again all of these but for the key parameter.
There are also some things that don't immediately come to mind, but that you could also test, if you have other parts of your system that rely on the algorithm having a particular property. That's what your test are for - they are meant to establish the contract between your component and its users. E.g. if you call encryptVariant on the same message with the same parameters twice, does it produce the same output or not? Should it produce the same output (or not)? (Many encryption algorithms are designed to produce different outputs, for security reasons.) So, in that light, think about what other parts of the system (callers) expect, and if you should capture that in your tests.
There might be a few more things to go through here, but let's say that that's about all the behavior that's "externally observable" through the Vigenere APIs. Everything else is an implementation detail as far as this particular test suite is concerned. Specific inputs and outputs are implementation dependent. If something depends on the internals and you think it's fairly likely to change at some point, ideally, you don't put it in your tests. At least, not in this set of tests. If you want to test the specifics of an algorithm that aren't covered here, have a separate set of tests for that (don't repeat the existing tests, but arrange things so that the implementation is covered by the two sets of tests).
What I outlined here is, more or less, what's known as behavioral testing. (I don't necessarily mean an approach like BDD, but the fundamental, bare bones notion of behavioral testing.) It's also related to property-based testing, as amon mentioned. The difference is less in philosophy, and more historical, and in terms of languages and tooling (not 100% sure, but I think one comes from math/comp.sci tradition, and the other from the Haskell community).
Now, if you're doing TDD, you might not have a fully formed vision of what the abstract behavior should be at the very beginning, so you might start by writing tests for a number of concrete examples and build up from there. That's fine.
But stop from time to time to see if there are any behavioral rules emerging, and try and re-express some of your tests in terms of those. Also, you might realize that some aspects of the original approach don't make sense in light of what you've learned since you've started, and that you need to reconceptualize things - and even delete some of the existing tests, and make new ones. That's fine. And sometimes, you'll feel as if your code is fighting your tests - e.g. the "arrange" part of the test might be quite long and involved, requiring a lot of setup before you can actually execute the test. That's your test telling you that your code knows too much about the other parts of the system - it's coupled to too many things and has deep dependencies. Such a test will easily break when something seemingly unrelated changes - but guess what, so will your code. In that case consider changing the design.
So, in summary: think of having behavioral tests as the goalpost, and try to move closer to it from time to time.