Speed is Distance

Speed is Distance | “I, the tiger, am so quick at running and turning that I can’t even be overtaken by a lightning bolt." Tom Leoni translation - Fior di Battaglia, Getty Museum
Though I have said that it is not your top speed that is important but how many speeds you have; being fast has its advantages.
To cultivate speed in your martial art you need to consider several aspects of training:

Power

The heart of speed is power. Power is your ability to exert a maximal force in as short a time as possible. Power is required to throw a ball, jump into the air, or run up a flight of stairs. Power is trained through exercises that require that you move a load (an external weight or your body) through a short and fast motion.
For a direct connection to your sword-work I recommend training with double weight swords or heavy sword like objects and moving them through your core cutting actions with speed and precision.
When approaching power exercises you need to make sure that the mechanics of your movement are correct and that you do not rush yourself to a weight that is overly challenging for you to move (i.e. you will strain yourself when doing it). Being able to complete 5 to 8 reps with proper form is optimal.
Be sure to consult a fitness expert to make sure you are using proper mechanics and using appropriate tools before embarking on this type of training.

Efficiency

Doing a movement in a mechanically precise fashion is not just essential to preventing injury but also to not wasting time in a movement. For an attack or defence to be as ‘fast’ as possible it must move directly from its starting position to its most effective destination position in a sound and effective fashion.
This is where slow leads to fast. If you jump to training at maximal speeds without first ensuring that your movement is as efficient as possible you’re wasting your time — literally.
Ideally every movement you develop is trained with expert consultation. Someone who can help guide you and coach you as you learn the movement. Once the movement is precisely defined gradually accelerate your practice of that motion and use self-discipline to not move faster than you are capable of doing the motion properly.

Timing & Reaction

In the Bolognese tradition just as in the practice of Bruce Lee, the element of surprise and choosing a moment to attack when your opponent is least prepared to defend creates the appearance of speed.
The Italian tradition we teach at Academie Duello provides an effective shorthand for the ideal moments for attack:

Prima Tempo - During an opponent’s step into your range.
Mezzo Tempo - While your opponent prepares to attack or otherwise changes position while in your range.
Dui-Tempi - Forcing your opponent to recover their weapon and at this moment striking them (such as beating their sword aside so they must bring it back online)
Contra Tempo - As your opponent begins to attack - as long as they’re attacking on your terms.